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Executive Summary 


The Massachusetts Office of District and School Turnaround (ODST) assists the 
Commissioner’s Districts (the 10 largest districts in the state) and schools within those districts. 
The assistance focuses on turning around the lowest performing schools in the district while 
building district capacity to support improvement in other district schools. The three major 
strategies examined in this study are as follows: 


" District Liaisons. Liaisons include ODST staff members who serve as project managers 
and coordinate support to the districts, overseeing implementation of the state’s strategy 
for school turnaround. 


« Priority Partners. Partners include external organizations that support turnaround efforts 
in four areas of support: maximizing learning time; the effective use of data; social, 
emotional, and health needs; and district systems of support. 


« School Redesign Grants (SRGs). SRGs are competitive funds that support turnaround 
efforts in persistently underperforming schools. The first year grants were awarded was 
2010-11. 


American Institutes for Research (AIR)' contracted with the Massachusetts Department of 
Elementary and Secondary Education (Massachusetts ESE) to conduct an evaluation of ODST 
assistance to Commissioner’s Districts and schools. This report summarizes the results from an 
impact analysis focused on the impact of the SRG receipt. SRGs are provided through federal 
School Improvement Grant (SIG) funding. Federal guidelines allow states flexibility in how they 
administer and monitor their distribution of SIG funds. The Massachusetts ESE allocates SIG 
funds as SRGs through a competitive application process. They commissioned this research to 
better understand how the process it designed has impacted or has failed to impact schools and 
how the process might be improved. This report answers the following research question: What 
is the impact of SRG receipt on student outcomes (e.g., academic achievement and attendance)?” 


Using a comparative interrupted time series (CITS) design, AIR researchers examined whether, 
when compared with non-SRG schools and controlling for selected background characteristics, 
students in SRG schools experienced better academic outcomes and attendance. 


Methods 


AIR used a CITS design to measure the impact of SRG receipt on student outcomes, including 
student achievement and attendance. The basic principle of CITS is that the SRG effect can be 
detected by comparing changes in the outcomes of SRG schools to changes in the outcomes in a 
comparison group during the same time period. This approach draws on information from both 
the treatment and comparison schools to estimate what performance in SRG schools would have 
been if the program had not been implemented. The deviation from this prediction is the 
estimated treatment effect of SRG receipt. 


' AIR is a behavioral and social science research organization founded in 1946. AIR carries out its work with strict 
independence, objectivity, and nonpartisanship. Its mission is to conduct and apply the best behavioral and social 
science research and evaluation to improve people’s lives, with a special emphasis on the disadvantaged. 
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The sample for this study included all students in cohorts | (beginning grant year of 2010-11), 

2 (beginning grant year of 2011-12), and 3 (beginning grant year of 2012-13) of the SRG 
schools, plus students within the same districts in comparison schools. Comparison schools were 
those in the same districts as the SRG schools but did not win an SRG. We used multilevel 
regression models to control for confounding factors (e.g., student characteristics), nesting of 
students within schools and years, and any changes in the given indicator across time that was 
not caused by the intervention itself. In addition, we controlled for student-level covariates (race, 
special education status, free- or reduced-price lunch status, and English language learner [ELL] 
status) and school-level factors (grant year, district, and whether the school was a high school) 
and allowed for baseline differences between schools. 


Findings 


The impact study found the following: 


= When considering prior achievement trends, students in the SRG schools performed 
better on the English language arts (ELA) and mathematics sections of the Massachusetts 
Comprehensive Assessment System (MCAS) compared with students in comparison 
schools. The effects were statistically significant after the first, second, and third years of 
SRG implementation on both the ELA and mathematics sections. 


= Positive SRG impacts on achievement were found for schools serving Grades 3-8 across 
all three years on both the ELA and mathematics sections. For schools serving Grade 10, 
positive impacts were found in Years | and 2 on the ELA section and in Year 1 on the 
mathematics section. 


= Positive impacts on both the ELA and mathematics sections were found in all three years 
of program implementation in Boston and for the first two years in Springfield and the 
other districts. 


= SRG receipt was associated with a decrease in the achievement gap on both the ELA and 
mathematics sections between ELL and non-ELL students in SRG schools compared 
with the change in the achievement gap between students in the comparison schools. 
These effects were found in all three years of program implementation on the ELA 
section and in the first two years on the mathematics section. The achievement gap also 
was decreased for students who had free- or reduced-price lunch status versus those who 
did not have such status in the SRG schools compared with similar students in 
comparison schools in the first year on the mathematics section but then increased in the 
second year. No changes were detected for students with special education status. 


= When considering prior achievement trends, students in the SRG schools appear to have 
slightly lower attendance rates compared with students in the comparison schools in the 
third year of SRG receipt. However, when examining these rates separately by school 
level, the effect appears to be negative for schools serving Grades 1-8 and positive for 
schools serving Grades 9—12 in the first year of program implementation. The effects 
appear negative for Years 2 and 3 in Boston, positive for Year 3 in Springfield, and 
positive for Year | in other districts. 


= SRG grant receipt was associated with a decrease in the attendance gap between students 
who were in special education versus who were not in special education in the SRG 
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schools compared with the change in the attendance gap between similar students in the 
comparison schools in all three years of program implementation. No changes in the 
attendance gap were detected for ELL versus non-ELL students and students with or 
without free- or reduced-price lunch status in the SRG schools compared with the 
comparison schools. 


Conclusion 


The results from this evaluation suggest that the disbursement of SIG grants in the process 
designed by ESE as SRGs have consistently positive effects on student academic achievement, 
particularly on standardized state assessments. Moreover, these results are generally robust 
across districts and school levels, and they are particularly strong for students who are ELLs. The 
results are neither strong nor consistent for attendance, suggesting that, overall, SRG receipt does 
not affect attendance. 
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I. Introduction 


It is widely known that many children in the United States attend low-performing schools— 
including schools that continue to fail their students even after years of reform interventions. To 
address these challenges, the 2010 Massachusetts Act Relative to the Achievement Gap gave 
additional powers to support turning around the lowest performing schools. Districts received 
greater power to intervene in the state’s lowest performing schools, and the Massachusetts ESE 
gained new and stronger policy tools to hold districts accountable for turning around the lowest 
performing schools. The resulting Framework for District Accountability and Assistance 
connected improvement efforts with districts and designated accountability levels (1-5) to 
schools based on their performance on and participation in standardized assessments. The 
highest performing schools are at Level 1, and the lowest performing schools are at Level 5. 


To support school turnaround in the Commissioner’s Districts, the Massachusetts ESE created 
ODST to provide assistance to these districts and schools within those districts to support school 
turnaround and enhance the district’s role in supporting struggling schools and ultimately turning 
them around. ODST assistance focuses on building capacity and the efforts of the participating 
districts to facilitate school turnaround for their highest need schools. The model hinges on a 
cycle of continuous improvement, a five-stage cycle that guides schools in developing a plan for 
improvement, implementing and monitoring the plan, evaluating success, analyzing data, and 
using the results to set the direction for future actions. This cycle continues as the school and the 
district build on past successes, learn from challenges, and refine the plan to make positive 
changes in the school. The three major strategies to support these efforts in the Commissioner’s 
Districts and build district capacity are as follows: 


« District Liaisons. ODST staff members serve as project managers and coordinate 
support to the districts, overseeing implementation of the state’s strategy for school 
turnaround). 


« Priority Partners. External partners support turnaround efforts regarding students’ 
social, emotional, and health needs. These partners maximize learning time, the effective 
use of data, and district systems of support. 


« SRGs. SRGs are competitive funds that support turnaround efforts in persistently 
underperforming schools. SRGs are provided through federal school improvement grant 
(SIG) funding. 


In many cases, the ODST assistance is aimed directly at the most struggling schools (usually 
Level 4 or Level 5), such as a priority partner providing tutoring services at a school or the SRG 
funding improvement efforts at a school. The assistance also can come in the form of improved 
support to the school from the district, which is working with a district liaison to better meet the 
needs of their Level 4 and Level 5 schools. Another goal of this assistance (through the district 
liaisons, priority partners, and SRGs) is to strengthen district capacity to guide and monitor 
school improvement. With this increased capacity, it is proposed that districts will be able to 
intervene in other schools in the district, particularly Level 3 schools, to further affect 
improvement in the district. Ultimately, the proposed outcomes of these efforts are as follows: 


= Student achievement improves in high need schools. 
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= District capacity to support school turnaround increases. 
= Achievement gaps narrow. 


= Performance improves in the district, particularly in the Level 3 and Level 4 schools, to 
“raise the floor” of district performance. 


AIR Evaluation 


AIR contracted with the Massachusetts ESE to conduct an evaluation of ODST assistance to 
Commissioner’s Districts and schools. The evaluation examined both implementation and impact 
of the district liaisons, priority partners, and the SRGs. This evaluation report provides the results 
from an impact analysis focused on assessing the effect of the SRG receipt on student outcomes. 


SRGs are provided through federal SIG funding. Federal guidelines allow states a great deal of 
flexibility in how they administer and monitor their distribution of SIG funds. The Massachusetts 
ESE commissioned this research to better understand how the process it designed has impacted 
or has failed to impact schools and how the process might be improved. The Massachusetts ESE 
allocates SIG funds as SRGs through a competitive application process. The request for 
proposals is designed according to specific indicators that research has demonstrated lead to 
school improvement, and the proposal evaluation criteria are developed to reflect these 
indicators. Proposals are individually reviewed and scored by three reviewers, who then meet 
with a trained facilitator to determine a final score, with individual readers allowed to dissent. 
Grants are awarded for a three-year period, with monitoring site visits conducted at the 
conclusion of each year. Schools are required to reapply following each concluded year to reflect 
on the successes and challenges of the year, address the findings of their monitoring site visits, 
and explain any changes in their plan moving forward. 


In the remainder of this report, we begin by describing the methodology used to conduct this 
CITS analysis. Then we present the findings, organized by outcome type and subgroup. We 
conclude with a discussion section focused on the implications of these findings and on issues 
that may warrant further study and attention. 
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II. Methods 


AIR used a CITS design to measure the impact of SRG grant receipt on student outcomes. The 
sample, outcome measures, and analyses are summarized in the following sections. 


Sample 


The sample for this study included all students in cohorts 1, 2, and 3 of the SRG schools, plus 
students within the same districts in comparison schools. Table | lists the SRG schools that 
comprised the sample for this analysis. Cohort 1 schools began implementation in the 2010-11 
school year, cohort 2 schools began implementation in the 2011—12 school year, and cohort 3 
schools began implementation in the 2012-13 school year. 


Table 1. Sample of the SRG Schools for the CITS Analysis 


District School Cohort 
Boston Blackstone School 1 
Boston Dearborn School 1 
Boston Elihu Greenwood School 1 
Boston The English High 1 
Boston Harbor School 1 
Boston Jeremiah E. Burke High School 2 
Boston John F. Kennedy School 1 
Boston John P. Holland School 1 
Boston Orchard Gardens School 1 
Boston Paul A. Dever School 1 
Boston William Monroe Trotter 1 
Brockton East Middle School 3 
Fall River John J. Doran 2 
Holyoke Morgan Elementary 2 
Holyoke Wm. J. Dean Vocational Technical High School 2 
Lowell Charlotte M. Murkland Elementary 2 
Lynn EJ Harrington 2 
Lynn William P. Connery Elementary School 3 
Springfield Alfred G. Zanetti 2 
Springfield Brightwood 2 
Springfield Chestnut Street Middle 2 
Springfield Elias Brookings 2 
Springfield Gerena 2 
Springfield High School of Commerce 3 
Springfield Homer Street 2 
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District School Cohort 
Springfield John F. Kennedy Middle 2 
Springfield M. Marcus Kiley Middle School 1 
Springfield White Street 2 
Worcester Chandler Elementary Community 2 
Worcester Burncoat Street Elementary School 3 
Worcester Union Hill School 2 


Comparison schools were those for which we had data and within the same districts as the SRG 
schools but did not win an SRG. This allowed us to compare schools within the same district 
(i.e., district fixed effects). 


Outcome Measures 


AIR examined the SRG impact on two outcomes: (1) student achievement and (2) student 
attendance. See Table Al and Table A2 in Appendix A for the distribution of the outcome 
measures. 


= Student Achievement. Student raw scores on MCAS in ELA and mathematics were the 
primary measure of student achievement. The MCAS raw scores were standardized 
across grade, years, and subject, and the standardized scores were pooled by subject. 


= Student Attendance. Student attendance was calculated as the total number of days a 
student attended school out of the total number of days they were enrolled at the school. 
Students who attended 30 days or less were removed from the analysis. 


Analysis 


AIR used a multilevel CITS method to measure the impacts of receiving an SRG on student 
outcomes. CITS is one of the strongest quasi-experimental designs that can be used when a 
comparison or control series can be constructed (Shadish, Cook, & Campbell, 2001). This 
method compares the outcomes of a treatment group and a comparison group after a treatment 
occurs relative to their baseline trends prior to program implementation to determine program 
impact. In doing so, this method measures program impact by comparing changes in the 
treatment group outcomes after program implementation to the changes in outcomes one would 
expect to find during the same time period in the absence of the program. The CITS analysis for 
the Massachusetts Urban District Assistance SRGs compares schools receiving SRGs” with other 
district schools serving approximately the same grade range of students.’ Differences in the 
baseline trends for SRG schools and the comparison schools preceding program implementation 
are compared with differences in average outcomes one, two, and three years following SRG 
receipt to demonstrate the extent to which a sharp discontinuity exists in the outcome measures 


* Data for UP Academy and Patrick F. Gavin Middle School were not available for 2012 and 2013 and thus have no 
postimplementation data; this school was therefore dropped from the analyses. The other 31 schools that received an 
SRG in the 2010, 2011, or 2012 school year cohorts are considered as the treatment groups in these analyses. 

* Schools are grouped into those serving any grades between Grades 1-8 (e.g., 1-8, 1-5, 56, and 7-8) and those 
serving Grades 9-12. 
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of students in SRG schools one, two, and three years following SRG receipt. (See Appendix A 
for a detailed description of the CITS model.) 


In addition to the overall model, AIR conducted several subgroup analyses to determine whether 
program impacts varied by population. The following subgroups were examined: 


= Three subgroups of districts: the Boston Public Schools (11 schools), the Springfield 
Public Schools (10 schools), and six other districts combined (10 schools) 


= High schools (four schools) and schools serving students in Grades 1-8 or 3-8 
(27 schools) 


= Special student populations identified by ELL status, free- or reduced-price lunch status, 
and special education status. 


Finally, the models included indicators for the school year, the district, whether the school is a 
high school, and students’ race and special student population classifications. See Table Al 
through Table A3 in Appendix A for the distribution of all the variables across time and between 
the SRG and comparison schools. 
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III. Findings 


This section describes the overall and subgroup analyses for each student outcome. 


Student Achievement 


Overall, after program implementation, when considering prior achievement trends, students in 
the SRG schools performed better on the MCAS ELA and mathematics sections compared with 
students in the comparison schools. Figure | through Figure 4 show the mean standardized raw 
scores for ELA and mathematics proficiency by grade and time point for the SRG schools and 
the comparison schools. All three cohorts of SRG schools serving Grades 1—8 show steady gains 
during this time period, closing the gap between their test scores and those of other schools 
within their district. Patterns are less clear among high schools receiving SRGs, with slight 
declines in test scores among all schools. 


Figure | and Figure 2 show that between 2007 and 2013, ELA and mathematics scores remained 
relatively flat for elementary and middle schools within those districts that were not receiving 
SRGs. These schools performed approximately one-half of a standard deviation lower, on 
average, compared with the mean performance of all schools in the state, and this performance 
remained stable. Schools receiving SRGs, however, were performing even worse in the years 
prior to receiving SRGs, more than a full standard deviation below the state mean for most 
measures. By 2013, these schools had substantially narrowed the gap between SRG and non- 
SRG schools. Tenth-grade scores show a less obvious pattern (Figure 3 and Figure 4), with both 
non-SRG and SRG schools within these districts showing slightly lower performance relative to 
the state mean during this time period. 
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Figure 1. Mean Standardized ELA MCAS Score With Schools Serving Grades 3-8 by 
School Year and Treatment Status 
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Figure 2. Mean Standardized Mathematics MCAS Score With Schools Serving Grades 3-8 
by School Year and Treatment Status 
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Figure 3. Mean Standardized ELA MCAS Score With Schools Serving Grade 10 by School 
Year and Treatment Status 
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Figure 4. Mean Standardized Mathematics MCAS Score With Schools Serving Grade 10 
by School Year and Treatment Status 
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Given prior trends in test scores one, two, and three years after receiving an SRG, students in 
SRG schools have ELA MCAS scores that are 0.16, 0.20, and 0.22 standard deviation higher, 
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respectively, than would be expected given test score changes in the comparison schools during 
the same time period (Figure 5). The effect sizes are 0.17, 0.21, and 0.22 standard deviation, 
respectively, for mathematics MCAS scores. (See Table B1 in Appendix B for the full results.) 
These results are substantial; considering that SRG schools were performing approximately half 
a standard deviation below other schools within their districts in 2007, the three-year results can 
be thought of as closing this gap by nearly half. 


Figure 5. MCAS ELA and Mathematics Achievement Score Effect Sizes by Year After 
Implementation 
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Subgroup Analyses 


Subgroup analyses were conducted based on district, school level, and special student population 
classification. The findings are summarized in the following subsections. 


District. The district subgroup analyses find a statistically significant positive impact of 
receiving a school redesign grant one and two years later for all three district subgroups; Boston, 
Springfield and Other (Appendix C, Table C1 through Table C3).* Significant positive impacts 
are found three years later only for Boston SRG schools on both the ELA and mathematics 
sections of MCAS. 


Grade Range. Subgroup analyses by grade range found a statistically significant positive impact 
of receiving an SRG one, two, and three years later in both ELA and mathematics for schools 
serving Grades 3—8. For SRG schools serving Grade 10, significant positive impacts one year 
later are found in ELA and mathematics, and two years later for ELA but not mathematics 
Appendix D, Table D1 and Table D2). It should be noted that because of the much smaller 


* Districts were divided into these three groups because Boston and Springfield each accounted for approximately 
one third of the total number of SRG schools, while the remaining third consisted of a small number of schools from 
the remaining districts that did not allow for individual district analyses. 
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number of high school students observed, it is more difficult to capture a statistically significant 
effect. 


Special Student Population Classification. The special student population analyses identified 
two Statistical differences in the changes in achievement gaps between the SRG and comparison 
schools (Appendix E, Table E1). First, the achievement gap between ELL and non-ELL students 
decreased on both the ELA and mathematics sections relative to the achievement gap between 
similar students in the comparison schools; these results were significant one, two, and three 
years after program implementation (Figure 6 and Figure 7). Second, the achievement gap 
between students receiving free- or reduced-priced lunch and those who do not decreased on 
both the ELA and mathematics sections relative to the achievement gap between similar students 
in the comparison schools one year later, but it increased two years later. No statistical 
differences were found for changes in MCAS scores for students with a special education 
classification. 


Figure 6. MCAS ELA Achievement Score Effect Sizes by English Language Learner (ELL) 
Status and Year After Implementation 
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Figure 7. MCAS Mathematics Achievement Score Effect Sizes by English Language 
Learner (ELL) Status and Year After Implementation 
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As shown in Figure 8, the attendance rate of students in the comparison schools serving 
Grades 1—8 remained relatively flat during the analysis window. Students in SRG schools seem 
to be increasing their attendance rate slightly during this time frame, but no obvious jumps or 
changes in slope are seen at the time of program implementation. Figure 9 shows that the 
attendance rate for high schools in the comparison group also is relatively flat as those for the 
SRG schools and shows no particular pattern (the small number of high schools in each cohort 
leads to greater random variation from year to year). After the introduction of SRGs, the only 
difference in the changes in student attendance rates between the students in the SRG and 
comparison schools relative to prior attendance rates is a slight negative effect in attendance 
three years after program implementation (Appendix B, Table B2). 


° The analyses for attendance were run in the arcsine metric to account for the fact that most attendance rates were 
close to one. Because the results remained unchanged, we chose to report the results without the arcsine 
transformation (i.e., in the original metric) for ease of interpretation. 
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Figure 8. Average Student Attendance Rate With Schools Serving Grades 1-8 by School 
Year and Treatment Status 
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Figure 9. Average Student Attendance Rate With Schools Serving Grades 9-12 by School 
Year and Treatment Status 


1,00 
1 
a 


0,95 
1 


Average Attendance Rate 
0.80 
i 


0.85 


0,80 
i 


American Institutes for Research Evaluation of Massachusetts School Redesign Grants—15 


Subgroup Analyses 


District. For the district subgroups, the effects of attendance rate appear to be negative two and 
three years later in Boston, positive three years later in Springfield, and positive in the first year 
in the other districts (Appendix C, Table C4 through Table C6). 


Grade Range. The subgroup analyses identified slight negative effects in attendance rate 
between the SRG and comparison schools in schools that serve Grades 1—8 and slight positive 
effects in the schools that serve Grades 9—12 in the first year (Appendix D, Table D3 and 
Table D4). No effects were detected in in the second and third years. 


Special Student Population Classification. The special student population analyses identified 
one consistent statistical difference in the changes in the attendance rate gaps between the SRG 
and comparison schools (Appendix E, Table E2). The attendance rate gap between special 
education and non-special education students decreased relative to the attendance rate gap 
between similar students in the comparison schools; these results were significant one, two, and 
three years after program implementation. 
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IV. Conclusion 


This report describes the findings from a quasi-experimental impact analysis that examined the 
extent to which student outcomes were associated with SRG receipt. It is one component of a 
larger evaluation that AIR conducted to assess the implementation and impact of ODST 
assistance to Commissioner’s Districts and schools. The outcomes examined here include student 
achievement and attendance. 


The results from this evaluation show that, when considering prior achievements trends, students 
in the SRG schools experienced greater gains on both the ELA and mathematics sections of 
MCAS compared with students in the comparison schools. These gains were particularly strong 
for students who are ELLs. Moreover, the gains were robust across districts and school levels 
and generally remained strong across all three years of program implementation. The results for 
attendance, on the other hand, were neither strong nor consistent, suggesting that SRG receipt 
does not generally affect attendance rates. 
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Appendix A. CITS Technical Approach 


AIR used a multilevel CITS model that accounts for nesting by means of district fixed effects, 
time random effects, and school random effects to determine whether SRGs had an impact on 
student outcomes (specifically student achievement and attendance rate) one, two, and three 
years after program implementation. The model is represented by the following equation: 


Y,, =4y + BGrant , + B,Time, + B,Grant Time, + B,PY1, + B,PY2,, + B,PY3, 
+/,Y11, + £,Y12, + J,Y13, + District, + HS, + Xjj+v,+uy +é&, 


In this model, Yj is the outcome measure (i.e., the standardized raw score or the attendance rate) 
for student i in school j at time t; Grant; is an indicator for school j that received an SRG (i.e., a 
treatment school); Time, is the outcome trend across time (years 2007—2013 are coded 1 through 
7, respectively); YJ/,, Y12,, and Y/3, are indicators for years 2011, 2012 and 2013; and PY/,, 
PY2,;, and PY3, are indicators for whether student j at time ft was in a school that had received an 
SRG one, two, and three years, respectively, after program implementation. In this model, each 
indicator for a student is coded as | if it applies to a student and 0 otherwise. For example, a 
student who has an outcome observed in a cohort 2 SRG school in 2012 would be coded 1 for 
Grant;, 1 for Y12, and 1 for PY/ (because 2012 is the first year in the post-SRG time period for 
schools in Cohort 2). Because Grant;Time, is an interaction between Grant; and Time, the 
student also would be coded as | multiplied by Time for Grant;Time, which allows for different 
pretreatment trends for SRG and non-SRG schools. District fixed effects and an indicator for 
whether the school is a high school are included in the model, allowing students in SRG schools 
to be compared only with students in non-SRG schools serving similar grade levels in the same 
district. The model also includes a set of student-level characteristics Xiz (i.e., race, gender, ELL 
status, free- or reduced-price lunch status, and special education status) that also may account for 
differences in the student outcomes. Random effects were included to account for school, cohort, 
and student effects by adding a random error term for each school (vj), time (u,), and student 


(€it)). 


Accounting for all student outcomes across time, the By, Bs, and B. coefficients in the model 
represent the difference in differences of outcomes between the SRG schools and the comparison 
schools prior to and one, two, and three years after SRG receipt, respectively (1.e., the one-year, 
two-year, and three-year posttreatment effects). In other words, these coefficients are the 
differences in outcomes for schools receiving SRGs one, two, and three years after receiving an 
SRG compared with their outcomes before receiving an SRG, subtracting out the difference in 
outcomes found in the comparison schools during the same time period. The B; coefficient 
compares the mean 2007 outcome score (i.e., the standardized MCAS raw score or the 
attendance rate) between students in treatment schools and comparison schools, the Bz coefficient 
represents the 2007—2010 trend in the outcome measure for the comparison schools, and the B3 
coefficient is the difference in the 2007—2010 outcome trend between the comparison and SRG 
schools. B7, Bg, and Bo are the differences in mean outcome for comparison schools in 2011, 2012, 
and 2013, respectively, compared with the 2007—2010 outcome trend (i.e., what would have been 
expected of SRG schools in these years in the absence of an SRG). 
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For the special populations subgroup analysis, an additional term was added to the model to 
determine whether receiving an SRG had an impact on student outcomes after program 
implementation: 


Y,,, =o + B.Grant, + £,Time, + B;Grant Time, + B,PY1, + BPY2, + B.PY3, + BY11, + BY12, 
+£Y13, + B,,Grant Subgroup, + f,,PY1, Subgroup, + B,,PY2,,Subgroup,, + PY 3, Subgroup, 
+District ,+ HS, + Xj +v,+uy +e, 


In this model, each subgroup analysis—ELL status, free- or reduced-price lunch status, and 
special education status—were run individually. PY/ Subgroup, through PY3,Subgroupi; were 
added to indicate whether there was a difference in the difference in the differences of a student 
in a subgroup versus not a subgroup in an SRG school versus a comparison school after program 
implementation. Therefore, the 81; through B13 coefficients represent the one-, two-, and three- 
year changes of the gap between students in a subgroup or not in the SRG schools versus the gap 
of students in the non-SRG schools after program implementation. GrantjSubgroupir 1s the 
preintervention time period difference in the differences between students in the subgroup versus 
not in a subgroup between SRG and non-SRG schools. Table Al through Table A3 show the 
distribution of the variables included in the models. 


American Institutes for Research Evaluation of Massachusetts School Redesign Grants—20 


Table Al. Mean ELA and Mathematics Scale Scores by Grade and Treatment Status 


ELA Mathematics 
N Raw Scores N Raw Scores 

S = E 5 E 5 E 5 E 5 

pe | - | | & | £ | ¢ | Gd | € 

a 5 a 5 a 5 a 5 
2007-08 1,370 10,404 26 (10) 32 (10) 1,406 10,490 21 (8) 26 (8) 
2008-09 1,390 10,211 25 (9) 30 (9) 1,372 10,304 22 (9) 26 (8) 
= 2009-10 1,383 10,556 24 (9) 30 (9) 1,417 10,636 20 (9) 26 (8) 
= 2010-11 1,413 10,277 26 (10) 31 (9) 1,437 10,454 21 (9) 25 (9) 
2011-12 1,277 10,315 29 (10) 32 (9) 1,310 10,456 24 (9) 26 (9) 
2012-13 1,376 10,655 29 (10) 31 (10) 1,414 10,766 24 (9) 26 (9) 
2013-14 1,412 10,889 29 (9) 31 (9) 1,442 10,986 25 (8) 26 (8) 
2007-08 1,348 10,149 38 (11) 46 (11) 1,381 10,196 25 (11) 33 (11) 
2008-09 1321 10,294 37 (12) 46 (11) 1,334 10,391 25 (11) 32 (11) 
5 2009-10 1,353 10,220 35 (11) 43 (11) L371 10,332 26 (11) 33 (12) 
2 2010-11 1,317 10,473 36 (11) 44 (12) 1,342 10,577 26 (11) 33 (11) 
a 2011-12 1,337 10,371 38 (12) 43 (12) L372 10,535 28 (12) 33 (12) 
2012-13 1,242 10,324 40 (12) 44 (11) 1,275 10,446 29 (11) 32 (11) 
2013-14 1,386 10,562 39 (12) 43 (12) 1,408 10,709 29 (11) 32 (11) 
2007-08 1,188 10,296 26 (10) 32 (10) 1,205 10,376 24 (12) 31 (12) 
2008-09 1,197 9,787 26 (9) 32 (9) 1,214 9,842 23 (12) 30 (12) 
= 2009-10 1,199 10,111 27 (9) 33 (9) 1,213 10,194 23 (12) 32 (13) 
2010-11 1,236 10,057 27 (10) 32 (10) 1,261 10,187 25 (12) 31 (12) 
2011-12 1,190 10,372 27 (9) 31 (10) 1,217 10,514 28 (11) 32 (12) 
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ELA Mathematics 
N Raw Scores N Raw Scores 

p | Steel |g : : : : : : : 

2 = = & 2 a 2 a 

a 5 a 5 a 5 a 5 
2012-13 1,176 10,238 28 (9) 31 (9) 1,199 10,362 28 (12) 31 (12) 
2013-14 1,071 9,799 28 (10) 31 (10) 1,096 9,881 29 (12) 32 (12) 
2007-08 1,292 9,918 28 (10) 32 (9) 1,303 9,984 23 (11) 30 (12) 
2008-09 1,328 9,601 28 (9) 33 (9) 1,349 9,687 25 (13) 33 (13) 
2009-10 1,213 9,304 29 (10) 34 (9) 1,229 9,376 25 (12) 32 (12) 
E 2010-11 1,369 9,533 27 (9) 32 (9) 1,387 9,633 24 (12) 32 (13) 
2011-12 1,401 9,450 27 (10) 32 (10) 1,419 9,571 25 (12) 31 (13) 
2012-13 1,315 9,677 28 (10) 31 (10) 1,340 9,762 27 (12) 32 (12) 
2013-14 1,331 9,620 28 (9) 31 (9) 1,372 9,744 26 (12) 31 (12) 
2007-08 1,478 10,237 42 (13) 46 (12) 1,494 10,303 23 (11) 28 (12) 
2008-09 1,501 9,841 41 (12) 46 (12) 1,528 9,917 22 (11) 28 (13) 
Ss 2009-10 1,499 9,529 40 (12) 46 (11) 1535 9,618 22( 12) 30 (13) 
: 2010-11 1,260 9,464 38 (12) 45 (12) 1,273 9,573 23 (12) 31 (13) 
ia 2011-12 1,287 9,800 39 (13) 45 (12) 1,293 9,922 24 (11) 31 (12) 
2012-13 1,166 9,865 39 (13) 44 (12) 1,188 9,986 24 (11) 30 (12) 
2013-14 1,165 10,150 39 (12) 44 (12) 1,192 10,250 24 (11) 30 (12) 
2007-08 1,506 10,516 29 (10) 33 (10) 1,526 10,515 20 (11) 26 (13) 
= 2008-09 1552 9,945 29 (10) 33 (9) Lor? 9,970 22 (11) 28 (12) 
iD 2009-10 1,522 9,765 29 (10) 34 (10) 1,551 9,824 2111) 28 (13) 
2010-11 1,432 9,491 29 (11) 33 (10) 1,442 9,597 24 (11) 31 (13) 
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2011-12 ieee) 9,479 29 (10) 34 (9) 1,224 9,541 24 (12) 31 (13) 
2012-13 1,159 9,765 29 (10) 34 (10) 1,173 9,822 22 (12) 29 (13) 
2013-14 1,146 9,945 29 (10) 33 (10) 1,169 10,062 24 (11) 29 (12) 
2007-08 737 10,124 41 (11) 47 (12) 711 9,885 27 (13) 34 (14) 
2008-09 694 9,937 39 (12) 48 (12) 633 9,723 24 (12) 34 (15) 

a 2009-10 726 9,834 40 (13) 49 (12) 682 9,707 23 (11) 32 (14) 

5 2010-11 648 9,771 37 (12) 47 (12) 648 9,660 23 (12) 34 (14) 

. 2011-12 556 9,705 41 (12) 49 (12) 543 9,585 25 (12) 33 (14) 
2012-13 631 9,269 39 (12) 48 (12) 639 9,212 24 (12) 35 (14) 
2013-14 500 9,406 Al (12) 50 (12) 488 9,340 23 (11) 33 (14) 


Note. Standard deviation is reported in parentheses. 


American Institutes for Research 


Evaluation of Massachusetts School Redesign Grants —23 


Table A2. Mean Attendance Rate by School Grade Range and Treatment Status 


Attendance Rate 
Grade School Year Mean 
Treatment Comparison Treatment Comparison 
2007 11,572 85,780 0.93 (0.08) 0.95 (0.06) 
2008 11,538 83,857 0.93 (0.08) 0.95 (0.06) 
2009 11,398 83,790 0.92 (0.08) 0.94 (0.06) 
7 2010 11,232 84,230 0.93 (0.07) 0.94 (0.06) 
2011 11,041 85,603 0.93 (0.07) 0.94 (0.06) 
2012 10,933 86,879 0.94 (0.07) 0.95 (0.06) 
2013 11,094 87,891 0.94 (0.06) 0.95 (0.06) 
2007 3172 34,591 0.85 (0.15) 0.90 (0.12) 
2008 2,722 33,995 0.84 (0.15) 0.90 (0.12) 
2009 pant ie 33,394 0.85 (0.16) 0.90 (0.12) 
m 2010 2,644 33,516 0.85 (0.15) 0.90 (0.12) 
2011 2,366 33,165 0.86 (0.13) 0.90 (0.12) 
2012 2,266 32,284 0.86 (0.14) 0.91 (0.12) 
2013 2,096 32,674 0.88 (0.12) 0.91 (0.12) 


Note. Standard deviation is reported in parentheses. 


American Institutes for Research 


Evaluation of Massachusetts School Redesign Grants —24 


Table A3. Student Demographics by Treatment Status and Year 


: : Free- or Reduced-Price 
, N Special Education lionchiecgecan ELLs 
> — S »—] S ~ S 1 = 
= 5 a 5 3 A = A 
) o = 5) Fs 5) . 5) . 
= E q E q E Fs E Fy 
B 3 E 5 E 5 E 5 E 
m 5 e 5 me 5 e 5 
2007-08 9,441 74,932 24.6% 19.4% 85.8% 72.9% 24.4% 15.3% 
2008-09 10,102 79,285 25.2% 19.8% 85.5% 71.5% 25.1% 16.2% 
2009-10 10,499 80,114 24.0% 20.0% 86.4% 73.7% 27.4% 18.2% 
2010-11 10,236 80,697 24.6% 20.0% 89.7% 76.4% 31.7% 22.6% 
2011-12 9,767 82,014 24.0% 20.2% 90.2% TI A% 34.0% 23.9% 
2012-13 9,435 82,017 22.5% 20.2% 89.7% 16.7% 34.5% 24.4% 
2013-14 9,374 83,408 21.3% 19.5% 91.2% 78.5% 32.6% 23.2% 
: z : Z Other or More 
b N Hispanic Black White Asian han Oue Race 
= 5 Z 5 2 5 2 5 Z 5 Z 5 Z 
5 : a : Ss g = E q E Ey 5 S 
is é F c F é F ‘ F c F c F 
Ee 6) z Oo = 6) a 6) e 6) a S 
2007-08 | 9441 74984 51.3% 33.1% 30.5% 26.1% 12.5% 29.6% 3.2% 8.9% 2.4% 2.2% 
2008-09 | 10104 79348 52.8% 34.5% 31.0% 25.8% 10.6% 28.5% 2.9% 8.8% 2.7% 2.3% 
2009-10 | 10516 80192 54.5% 35.5% 30.2% 25.6% 9.0% 27.5% 3.1% 8.9% 3.0% 2.5% 
2010-11 | 10266 80890 55.3% 36.5% 29.3% 25.0% 8.9% 26.6% 3.4% 8.9% 2.8% 2.7% 
2011-12 | 9767 82014 56.9% 37.4% 28.5% 24.7% 8.3% 26.0% 3.5% 9.1% 2.6% 2.8% 
2012-13 | 9447 82064 56.7% 37.4% 28.4% 25.1% 8.4% 25.4% 3.7% 8.9% 2.7% 3.1% 
2013-14 | 9374 83408 56.4% 38.3% 28.0% 24.4% 8.7% 24.7% 3.9% 9.1% 2.8% 3.3% 
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Appendix B. CITS Outcomes 


Table B1 and Table B2 present the CITS outcomes for student achievement and attendance rate, 
respectively. For each model, the coefficients on Grant x Post Year 1, Grant x Post Year 2, and 
Grant x Post Year 3 represent the overall effects of an SRG one, two, and three years after 
receiving the grant. These effects represent the changes in the outcomes of students in the SRG 
schools after program implementation compared with changes in outcomes of students in the 
comparison schools while controlling for other student-level demographics (i.e., special 
education status, free- or reduced-price lunch status, ELL status, gender, and race). Student 
achievement outcomes are standardized; thus effect sizes should be interpreted as standard 
deviation changes. Attendance outcomes are presented as rates; thus effect sizes should be 
interpreted as percentage point changes. 


Because of the cohort structure of program implementation (cohort | received an SRG at the 
start of the 2011 school year, cohort 2 at the start of the 2012 school year, and cohort 3 at the 
start of the 2013 school year) and outcome data were available only through 2013, the effects of 
receiving a grant one year later (the first school year after receiving the grant) are estimated for 
all schools, the effects of receiving a grant two years later are estimated for schools in cohorts 1 
and 2, and the effects of receiving a grant three years later are estimated only for schools in 
cohort 1. 
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Table B1. CITS Outcomes: Student Achievement 


ELA Mathematics 
Grant (6) -0.29** (0.07) -0.28** (0.07) 
Time (B) 0.02** (0.00) 0.02** (0.01) 


Grant x Time (B53) 


-0.01* (0.00) 


0.00 (0.00) 


Grant x Post Year | (B4) 


0.16** (0.01) 


0.17** (0.01) 


Grant x Post Year 2 (Bs) 


0.20** (0.02) 


0.21** (0.02) 


Grant x Post Year 3 (B¢) 


0.22** (0.02) 


0.22** (0.02) 


Year 2011 (By) -0.01 (0.02) 0.00 (0.02) 
Year 2012 (Bs) -0.05* (0.02) -0.02 (0.03) 
Year 2013 (Bo) -0.07** (0.02) -0.04 (0.03) 


High school 


0.10** (0.01) 


0.18** (0.01) 


Special education 


-1.01** (0.00) 


-0.86** (0.00) 


Free- or reduced-price lunch program -0.25** (0.00) -0.22** (0.00) 
English language learner -0.79** (0.00) -0.54** (0.00) 
Hispanic -0.22** (0.00) -0.26** (0.00) 
Black -0.25** (0.00) -0.37** (0.00) 
Asian/other 0.07** (0.00) 0.21** (0.00) 
Male -0.18** (0.00) 0.07** (0.00) 
Brockton -0.10 (0.07) -0.11 (0.07) 
Fall River -0.17** (0.07) -0.32** (0.07) 
Holyoke -0.42** (0.10) -0.39** (0.10) 
Lowell 0.02 (0.07) -0.06 (0.08) 
Lynn 0.08 (0.07) 0.02 (0.07) 
Springfield -0.07 (0.06) -0.15** (0.06) 
Worcester 0.15** (0.06) 0.05 (0.06) 
Constant (Bo) 0.09** (0.03) -0.01 (0.03) 
Time random-effects parameters 0.00 (0.01) 0.00 (0.01) 
School random-effects parameters 0.12 (0.34) 0.12 (0.34) 
Residual random-effects parameters 0.81 (0.90) 0.79 (0.89) 


Note. Standard errors are presented in parentheses. 


* p< .05. ** p< .01. 
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Table B2. CITS Outcomes: Attendance Rate 


Attendance Rate 
Grant (6) -0.006 (0.007) 
Time (B) -0.001 (0.000) 
Grant x Time (f3) 0.002** (0.000) 


Grant x Post Year 1 (B,) 


0.000 (0.001) 


Grant x Post Year 2 (Bs) 


-0.001 (0.001) 


Grant x Post Year 3 (Bs) 


-0.004* (0.002) 


Year 2011 (By) 


0.002 (0.001) 


Year 2012 (Bs) 


0.005** (0.002) 


Year 2013 (Bo) 


0.006** (0.002) 


High school 


-0.025** (0.001) 


Special education 


-0.019** (0.000) 


Free- or reduced-price lunch program 


-0.014** (0.000) 


English language learner 


0.010** (0.000) 


Hispanic -0.009** (0.000) 
Black 0.009** (0.000) 
Asian/other -0.015** (0.000) 
Male 0.000* (0.000) 
Brockton 0.003 (0.007) 
Fall River -0.009 (0.007) 
Holyoke -0.009 (0.010) 
Lowell 0.006 (0.008) 
Lynn 0.012 (0.007) 
Springfield -0.006 (0.006) 
Worcester 0.027** (0.006) 


Constant (Bo) 


0.941** (0.003) 


Time random-effects parameters 0.000 (0.001) 
School random-effects parameters 0.001 (0.037) 
Residual random-effects parameters 0.006 (0.075) 


Note. Standard errors are presented in parentheses. 


* p< 05. ** p< .01. 
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Appendix C. CITS Outcomes by District 


Table C1 through Table C6 show the CITS outcomes for student achievement and attendance 
rate separately for three groups of districts: the Boston Public Schools, the Springfield Public 
Schools, and six other districts combined. Separate analyses were conducted in this way because 
Boston and Springfield each comprise approximately one third of the total number of schools 
receiving an SRG. Conducting analyses separately by district allows for determining whether 
receiving an SRG had a statistically significant effect specifically for schools in Boston, 
Springfield, and the other six districts. This is potentially important because with the large 
proportion of schools in Boston and Springfield, the main results are driven largely by the 
schools in these two districts. It is therefore possible that the main results might find a 
statistically significant program effect, but the subgroup analyses reveal that the impact is 
statistically significant only for schools in one of these district subgroups. The reverse also is 
possible; the main findings may be null, whereas one or more of the groups of schools has 
statistically significant effects. For each model, the coefficients on the Grant x Post Year 1, 

Grant x Post Year 2, and Grant x Post Year 3 represent the effects of receiving an SRG one, two, 
and three years after receiving the grant for all schools within the district(s) included in the 
subgroup. Because some districts did not have schools in all cohorts, not all subgroups have effects 
for all postimplementation years. 
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Table C1. CITS Outcomes: Boston Student Achievement 


ELA Mathematics 
Grant (6) -0.42** (0.11) -0.39** (0.11) 
Time (B) 0.02* (0.01) 0.01 (0.01) 
Grant x Time (f3) 0.01 (0.01) 0.02* (0.01) 


Grant x Post Year 1 (B4) 


0.18** (0.03) 


0.26** (0.03) 


Grant <x Post Year 2 (Bs) 


0.29** (0.04) 


0.28** (0.03) 


Grant x Post Year 3 (B.¢) 


0.25** (0.04) 


0.27** (0.04) 


Year 2011 (By) -0.02 (0.03) -0.01 (0.05) 
Year 2012 (Bs) -0.09** (0.03) -0.03 (0.06) 
Year 2013 (Bo) -0.10* (0.04) -0.06 (0.07) 


High school 


0.09** (0.01) 


0.18** (0.01) 


Special education 


-0.94** (0.01) 


-0.78** (0.01) 


Free- or reduced-price lunch program 


-0.16** (0.01) 


-0.11** (0.01) 


English language learner 


-0.76** (0.01) 


-0.46** (0.01) 


Hispanic -0.21** (0.01) -0.26** (0.01) 
Black -0.32** (0.01) -0.41** (0.01) 
Asian/other 0.05** (0.01) 0.30** (0.01) 
Male -0.20** (0.00) 0.04** (0.00) 
Constant (B9) 0.05 (0.04) -0.10* (0.05) 
Time random-effects parameters 0.00 (0.02) 0.00 (0.03) 
School random-effects parameters 0.12 (0.34) 0.12 (0.35) 
Residual random-effects parameters 0.80 (0.89) 0.77 (0.88) 


Note. Standard errors are presented in parentheses. 


* p< 05. ** p< 01. 
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Table C2. CITS Outcomes: Springfield Student Achievement 


ELA Mathematics 
Grant (6) -0.26 (0.14) -0.30 (0.16) 
Time (B) -0.00 (0.01) -0.01 (0.01) 
Grant x Time (f3) -0.01* (0.01) 0.00 (0.01) 


Grant x Post Year 1 (B4) 


0.07** (0.02) 


0.09** (0.02) 


Grant <x Post Year 2 (Bs) 0.06* (0.03) 0.06* (0.03) 
Grant x Post Year 3 (B.¢) 0.00 (0.05) -0.03 (0.04) 
Year 2011 (B7) 0.03 (0.02) 0.03 (0.02) 
Year 2012 (Bs) 0.04 (0.02) 0.04 (0.02) 
Year 2013 (Bo) 0.05 (0.03) 0.09** (0.03) 


High school 


0.11** (0.04) 


0.28** (0.04) 


Special education 


-0.98** (0.01) 


-0.77** (0.01) 


Free- or reduced-price lunch program 


-0.34** (0.01) 


-0.29** (0.01) 


English language learner 


-0.84** (0.01) 


-0.56** (0.01) 


Hispanic -0.28** (0.01) -0.34** (0.01) 
Black -0.23** (0.01) -0.38** (0.01) 
Asian/other 0.03* (0.02) 0.07** (0.01) 
Male -0.16** (0.01) 0.09** (0.01) 
Constant (B9) 0.17** (0.06) -0.04 (0.07) 
Time random-effects parameters 0.00 (0.01) 0.00 (0.01) 
School random-effects parameters 0.15 (0.39) 0.19 (0.44) 
Residual random-effects parameters 0.85 (0.92) 0.77 (0.88) 


Note. Standard errors are presented in parentheses. 


* p< 05. ** p< 01. 
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Table C3. CITS Outcomes: Other Districts Student Achievement 


ELA Mathematics 
Grant (61) -0.31** (0.11) -0.27** (0.10) 
Time (2) 0.03** (0.00) 0.02** (0.00) 
Grant x Time (£3) 0.01 (0.01) 0.01* (0.01) 


Grant x Post Year 1 (B,) 


0.24** (0.03) 


0.15** (0.03) 


Grant x Post Year 2 (Bs) 


0.30** (0.03) 


0.24** (0.03) 


Grant x Post Year 3 (fs) 


Year 2011 (By) -0.01 (0.01) -0.01 (0.01) 
Year 2012 (Bs) -0.04* (0.01) -0.02 (0.02) 
Year 2013 (Bo) -0.08** (0.02) -0.05* (0.02) 


High school 


0.14** (0.03) 


0.11 (0.03) 


Special education 


-1.07** (0.00) 


-0.94** (0.00) 


Free- or reduced-price lunch program 


-0.29** (0.00) 


-0.28** (0.00) 


English language learner 


-0.81** (0.00) 


-0.59** (0.00) 


Hispanic -0.21** (0.00) -0.23** (0.00) 
Black -0.20** (0.00) -0.31** (0.01) 
Asian/other 0.07** (0.01) 0.18** (0.01) 
Male -0.18** (0.00) 0.08** (0.00) 
Fall River -0.04 (0.08) -0.19* (0.08) 
Holyoke -0.30** (0.11) -0.27** (0.10) 
Lowell 0.15 (0.09) 0.08 (0.08) 
Lynn 0.20* (0.09) 0.14 (0.08) 
Worcester 0.26** (0.08) 0.17* (0.07) 
Constant (Bo) -0.03 (0.06) -0.10 (0.06) 
Time random-effects parameters 0.00 (0.01) 0.00 (0.01) 
School random-effects parameters 0.10 (0.32) 0.09 (0.30) 
Residual random-effects parameters 0.81 (0.90) 0.80 (0.89) 


Note. Standard errors are presented in parentheses. 


* p< 05. ** p < 01. 
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Table C4. CITS Outcomes: Boston Attendance Rate 


Attendance Rate 
Grant (6) -0.007 (0.011) 
Time (B) -0.002 (0.001) 
Grant x Time (f3) 0.002** (0.001) 


Grant x Post Year 1 (B,) 


-0.002 (0.002) 


Grant x Post Year 2 (Bs) 


-0.007** (0.002) 


Grant x Post Year 3 (fs) 


-0.07* (0.003) 


Year 2011 (By) 0.004 (0.004) 
Year 2012 (Bs) 0.006 (0.005) 
Year 2013 (Bo) 0.006 (0.006) 


High school 


-0.026** (0.001) 


Special education 


-0.018** (0.000) 


Free- or reduced-price lunch program 


-0.009** (0.000) 


English language learner 


0.017** (0.000) 


Hispanic -0.006** (0.001) 
Black 0.010** (0.001) 
Asian/other 0.024** (0.001) 
Male -0.001** (0.000) 


Constant (B9) 


0.938** (0.004) 


Time random-effects parameters 


0.000 (0.002) 


School random-effects parameters 


0.001 (0.037) 


Residual random-effects parameters 


0.007 (0.085) 


Note. Standard errors are presented in parentheses. 


* p< 05. ** p < OL. 
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Table C5. CITS Outcomes: Springfield Attendance Rate 


Attendance Rate 
Grant (6) 0.001 (0.017) 
Time (B) 0.003** (0.001) 
Grant x Time (£3) 0.001 (0.000) 


Grant x Post Year 1 (B,) 


-0.003 (0.002) 


Grant x Post Year 2 (Bs) 


0.001 (0.002) 


Grant x Post Year 3 (f«) 


0.013** (0.004) 


Year 2011 (By) -0.002 (0.003) 
Year 2012 (Bs) 0.004 (0.004) 
Year 2013 (Bo) 0.004 (0.005) 


High school 


-0.014** (0.002) 


Special education 


-0.016** (0.001) 


Free- or reduced-price lunch program 


-0.018** (0.001) 


English language learner 


-0.004** (0.001) 


Hispanic -0.018** (0.001) 
Black 0.001 (0.001) 

Asian/other 0.012** (0.001) 
Male 0.001** (0.000) 


Constant (B9) 


0.929** (0.008) 


Time random-effects parameters 


0.000 (0.002) 


School random-effects parameters 


0.002 (0.049) 


Residual random-effects parameters 


0.008 (0.087) 


Note. Standard errors are presented in parentheses. 


* p< 05. ** p< 01. 
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Table C6. CITS Outcomes: Other Districts Attendance Rate 


Attendance Rate 
Grant (6) 0.002 (0.011) 
Time (B) -0.001* (0.000) 
Grant x Time (£3) 0.000 (0.000) 


Grant x Post Year 1 (B,) 


0.004*** (0.002) 


Grant x Post Year 2 (Bs) 


0.003 (0.002) 


Grant x Post Year 3 (fs) 


Year 2011 (By) 


0.002 (0.001) 


Year 2012 (Bs) 


0.006** (0.001) 


Year 2013 (Bo) 


0.007** (0.002) 


High school 


-0.030** (0.002) 


Special education 


-0.019** (0.000) 


Free- or reduced-price lunch program 


-0.017** (0.000) 


English language learner 


0.008** (0.000) 


Hispanic -0.007** (0.000) 
Black 0.014** (0.000) 
Asian/other 0.012** (0.000) 
Male 0.000 (0.000) 
Fall River -0.011 (0.008) 
Holyoke -0.010 (0.011) 
Lowell 0.006 (0.009) 
Lynn 0.011 (0.009) 
Worcester 0.025** (0.008) 


Constant (Bo) 


0.946** (0.006) 


Time random-effects parameters 0.000 (0.001) 
School random-effects parameters 0.001 (0.033) 
Residual random-effects parameters 0.004 (0.063) 


Note. Standard errors are presented in parentheses. 


* p< .05. ** p< .01. 
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Appendix D. CITS Outcomes by Grades Served 


Table D1 through Table D4 the CITS outcomes for student achievement and attendance rate 
separately for schools serving any grades between kindergarten and Grade 8 (e.g., schools 
serving Grades 1-8, 1-5, 5—8, and 7-8) and those serving Grades 9-12. Conducting analyses 
separately by grade range served allows for determining whether receiving an SRG had a 
statistically significant effect separately among schools serving Grades 1—8 and those serving 
Grades 9-12. This is potentially important because it is possible that the impact of SRGs differ 
depending on whether the school serves primary or high school students. For each model, the 
coefficients on the Grant x Post Year 1, Grant x Post Year 2, and Grant x Post Year 3 represent 
the effects of receiving an SRG one, two, and three years after receiving a grant for all schools in 
the grade range subgroup. 
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Table D1. CITS Outcomes: Schools Serving Grades 3-8—Student Achievement 


ELA Mathematics 
Grant (6) -0.35** (0.06) -0.35** (0.06) 
Time (B) 0.02** (0.01) 0.02* (0.01) 
Grant x Time (f3) -0.00 (0.00) 0.00 (0.00) 


Grant x Post Year 1 (B4) 


0.16** (0.02) 


0.17** (0.02) 


Grant <x Post Year 2 (Bs) 


0.19** (0.02) 


0.21** (0.02) 


Grant x Post Year 3 (B.¢) 


0.21** (0.02) 


0.20** (0.02) 


Year 2011 (By) -0.02 (0.02) -0.01 (0.03) 
Year 2012 (Bs) -0.05* (0.02) -0.03 (0.03) 
Year 2013 (Bo) -0.08** (0.03) -0.04 (0.04) 


Special education 


-1.01** (0.00) 


-0.86** (0.00) 


Free- or reduced-price lunch program 


-0.27** (0.00) 


-0.25** (0.00) 


English language learner 


-0.74** (0.00) 


-0.52 (0.00) 


Hispanic -0.21** (0.00) -0.25** (0.00) 
Black -0.25** (0.00) -0.36** (0.00) 
Asian/other 0.08** (0.01) 0.20** (0.01) 
Male -0.19** (0.00) 0.07** (0.00) 
Brockton -0.16* (0.07) -0.13 (0.07) 
Fall River -0.22** (0.06) -0.36** (0.06) 
Holyoke -0.53** (0.09) -0.48** (0.09) 
Lowell -0.07 (0.07) -0.13 (0.07) 
Lynn 0.05 (0.07) 0.03 (0.07) 
Springfield -0.06 (0.05) -0.12* (0.05) 
Worcester 0.07 (0.06) -0.01 (0.06) 
Constant (Bo) 0.18** (0.03) 0.08* (0.04) 
Time random-effects parameters 0.00 (0.01) 0.00 (0.02) 
School random-effects parameters 0.09 (0.30) 0.09 (0.30) 
Residual random-effects parameters 0.81 (0.90) 0.80 (0.89) 


Note. Standard errors are presented in parentheses. 


* p< 05. ** p< .01. 
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Table D2. CITS Outcomes: Schools Serving Grade 10—Student Achievement 


ELA Mathematics 
Grant (6) -0.13 (0.19) -0.05 (0.21) 
Time (B) -0.00 (0.01) 0.00 (0.00) 


Grant x Time (3) 


-0.03** (0.01) 


-0.05** (0.01) 


Grant x Post Year 1 (B4) 


0.16** (0.05) 


0.17#* (0.05) 


Grant <x Post Year 2 (Bs) 0.19** (0.06) 0.10 (0.06) 
Grant x Post Year 3 (B.¢) 0.12 (0.11) 0.09 (0.11) 
Year 2011 (B7) 0.04 (0.02) 0.02 (0.02) 
Year 2012 (Bs) 0.02 (0.03) 0.06** (0.02) 
Year 2013 (Bo) 0.02 (0.04) -0.05* (0.02) 


Special education 


-1.06** (0.01) 


-0.88** (0.01) 


Free- or reduced-price lunch program 


-0.16** (0.01) 


-0.10** (0.01) 


English language learner 


-1.28** (0.01) 


-0.66 (0.01) 


Hispanic -0.28** (0.01) -0.33** (0.01) 
Black -0.28** (0.01) -0.40** (0.01) 
Asian/other -0.03* (0.01) 0.22** (0.01) 
Male -0.15** (0.01) -0.09** (0.01) 
Brockton -0.30 (0.18) -0.49** (0.19) 
Fall River -0.67** (0.22) -0.82** (0.23) 
Holyoke -0.18 (0.22) -0.32 (0.23) 
Lowell -0.05 (0.28) -0.32 (0.30) 
Lynn -0.14 (0.16) -0.43* (0.17) 
Springfield -0.33* (0.14) -0.51** (0.14) 
Worcester 0.31* (0.15) 0.08 (0.16) 
Constant (Bo) 0.01 (0.06) -0.03 (0.06) 
Time random-effects parameters 0.00 (0.01) 0.00 (0.00) 
School random-effects parameters 0.13 (0.36) 0.15 (0.38) 
Residual random-effects parameters 0.77 (0.88) 0.70 (0.84) 


Note. Standard errors are presented in parentheses. 


* p< 05. ** p< .01. 
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Table D3. CITS Outcomes: Schools Serving Grades 1-8—Attendance Rate 


Attendance Rate 
Grant (6) -0.007 (0.006) 
Time (B) -0.001 (0.001) 
Grant x Time (f3) 0.001** (0.000) 


Grant x Post Year 1 (B,) 


-0.002** (0.001) 


Grant x Post Year 2 (Bs) 


0.002 (0.001) 


Grant x Post Year 3 (fs) 


-0.002 (0.001) 


Year 2011 (By) 


0.003 (0.002) 


Year 2012 (Bs) 


0.006** (0.002) 


Year 2013 (Bo) 


0.006* (0.003) 


Special education 


-0.016** (0.000) 


Free- or reduced-price lunch program 


-0.014** (0.000) 


English language learner 


0.008** (0.000) 


Hispanic -0.006** (0.000) 
Black 0.010** (0.000) 
Asian/other 0.014** (0.000) 
Male -0.001** (0.000) 
Brockton -0.001 (0.006) 
Fall River -0.016** (0.006) 
Holyoke -0.016 (0.009) 
Lowell -0.002 (0.007) 
Lynn 0.004 (0.006) 
Springfield -0.005 (0.005) 
Worcester 0.018** (0.005) 


Constant (Bo) 


0.952** (0.003) 


Time random-effects parameters 


0.000 (0.001) 


School random-effects parameters 


0.001 (0.029) 


Residual random-effects parameters 


0.003 (0.057) 


Note. Standard errors are presented in parentheses. 


* p< 05. ** p< .01. 
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Table D4. CITS Outcomes: Schools Serving Grades 9-12—Attendance Rate 


Attendance Rate 
Grant (6) -0.006 (0.026) 
Time (B) 0.001** (0.000) 
Grant x Time (f3) 0.003** (0.001) 


Grant x Post Year 1 (B,) 


0.014** (0.003) 


Grant x Post Year 2 (Bs) 


-0.010* (0.004) 


Grant x Post Year 3 (fs) 


0.006 (0.006) 


Year 2011 (By) 


-0.001 (0.001) 


Year 2012 (Bs) 


0.003* (0.001) 


Year 2013 (Bo) 


0.005** (0.001) 


Special education 


-0.028** (0.001) 


Free- or reduced-price lunch program 


-0.015** (0.001) 


English language learner 


0.017** (0.001) 


Hispanic -0.017** (0.001) 
Black 0.007** (0.001) 
Asian/other 0.018** (0.001) 
Male 0.002** (0.000) 
Brockton -0.040 (0.025) 
Fall River -0.074* (0.032) 
Holyoke -0.049 (0.029) 
Lowell -0.024 (0.039) 
Lynn 0.010 (0.023) 
Springfield -0.026 (0.018) 
Worcester 0.055* (0.022) 


Constant (Bo) 


0.886** (0.008) 


Time random-effects parameters 


0.000 (0.000) 


School random-effects parameters 


0.003 (0.053) 


Residual random-effects parameters 


0.012 (0.110) 


Note. Standard errors are presented in parentheses. 


* p< 05. ** p< .01. 


American Institutes for Research 


Evaluation of Massachusetts School Redesign Grants —40 


Appendix E. CITS Outcomes by Special Population 


Table El and Table E2 show the CITS outcomes for student achievement and attendance rate for 
three special populations of students: ELLs, special education, and free- or reduced-price lunch. 
The coefficients on the Grant x Post Year 1, Grant x Post Year 2, and Grant x Post Year 3 
represent the overall effects on outcomes of a school receiving an SRG one, two, and three years 
after receiving a grant. These effects represent the changes in the outcomes of students in the SRG 
schools after program implementation compared with changes in the outcomes of students in the 
comparison schools while controlling for other student-level demographics (i.e., special education 
status, free- or reduced-price lunch status, ELLs, gender, and race). Interactions between being in a 
treatment school and being a special population student are included in these models to determine 
whether being in a school that received an SRG had differential impact for special populations of 
students. The coefficients on Subgroup x Post Year 1, Subgroup x Post Year 2, and Subgroup x 
Post Year 3 represent the differences in the effect of being in a school that received an SRG 
between students who are and are not of the special population (i.e., the difference-in-difference- 
in-difference). 
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Table E1. Regression Outcomes for Special Populations Within the SRG and Comparison Schools: Student Achievement 


English Language Learners 


Free-or Reduced-Price Lunch 


Special Education 


ELA 


Mathematics 


ELA 


Mathematics 


ELA 


Mathematics 


Grant (B;) -0.25** (0.07) | -0.27** (0.01) | -0.26** (0.07) | -0.30** (0.07) | -0.31** (0.07) | -0.30** (0.07) 
Time (B2) 0.02** (0.00) | 0.02** (0.01) | 0.02** (0.00) | 0.02** (0.01) | 0.02** (0.00) | 0.02** (0.01) 
Grant < Time (B3) -0.00 (0.00) 0.00 (0.00) -0.01* (0.00) 0.00 (0.00) -0.01* (0.00) 0.00 (0.00) 

Grant x Post Year 1 (B4) 0.10** (0.02) | 0.14** (0.02) 0.09* (0.04) 0.10** (0.04) | 0.17** (0.02) | 0.18** (0.02) 


Grant x Post Year 2 (Bs) 


0.10** (0.02) 


0.17** (0.02) 


0.22** (0.04) 


0.30** (0.04) 


0.19** (0.02) 


0.21** (0.00) 


Grant x Post Year 3 (B¢) 


0.16** (0.03) 


0.20** (0.03) 


0.22% (0.05) 


0.18** (0.05) 


0.22** (0,03) 


0.22** (0.02) 


Grant x Subgroup (B10) -0.20** (0.01) | -0.04** (0.01) | -0.03* (0.01) 0.02 (0.01) 0.06** (0.01) | 0.10** (0.01) 
Grant x Post Year 1 x Special we ‘hip x ; ; 

Population (61) 0.21** (0.02) | 0.10** (0.02) 0.07 (0.04) 0.08* (0.04) 0.01 (0.03) 0.02 (0.03) 
Grant x Post Year 2 x Special be ie ‘ : Sa 

Population (B,s) 0.30** (0.03) | 0.13** (0.02) 0.02 (0.04) 0.10** (0.04) 0.04 (0.03) 0.03 (0.03) 
Grant x Post Year 3 x Special ee ; 

Population (B13) 0.17** (0.04) 0.04 (0.04) 0.00 (0.05) 0.05 (0.05) 0.01 (0.04) 0.04 (0.04) 
Year 2011 (B,) -0.01 (0.02) 0.00 (0.02) -0.01 (0.02) 0.00 (0.02) -0.01 (0.02) 0.00 (0.02) 
Year 2012 (Bg) -0.05* (0.02) -0.02 (0.03) -0.04* (0.02) -0.02 (0.03) -0.05* (0.02) -0.02 (0.03) 
Year 2013 (Bo) -0.07** (0.02) -0.04 (0.03) -0.07** (0.02) -0.04 (0.03) -0.07** (0.02) -0.04 (0.03) 
Special education -1.01** (0.00) | -0.86** (0.00) | -1.01** (0.00) | -0.86** (0.00) | -1.02** (0.00) | -0.87** (0.00) 
Free- or reduced-price lunch -0.25** (0.00) | -0.22** (0.00) | -0.25** (0.00) | -0.23** (0.00) | -0.25** (0.00) | -0.22** (0.00) 
English language learner -0.78** (0.00) -0.54 (0.00) -0.79** (0.00) | -0.54** (0.00) | -0.79** (0.00) | -0.54** (0.00) 
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English Language Learners 


Free-or Reduced-Price Lunch 


Special Education 


ELA Mathematics ELA Mathematics ELA Mathematics 
Hispanic -0.22** (0.00) -0.26 (0.00) -0.22** (0.00) | -0.26** (0.00) | -0.22** (0.00) | -0.26** (0.00) 
Black -0.25** (0.00) | -0.37** (0.00) | -0.25** (0.00) | -0.37** (0.00) | -0.25** (0.00) | -0.37** (0.00) 
Asian/other 0.07** (0.00) | 0.21** (0.00) | 0.07** (0.00) | 0.21** (0.00) | 0.07** (0.01) | 0.21** (0.00) 
Brockton -0.10 (0.07) -0.11 (0.07) -0.10 (0.07) -0.11 (0.07) -0.10 (0.07) -0.11 (0.07) 
Fall River -0.17** (0.07) | -0.32** (0.07) | -0.17** (0.07) | -0.32** (0.07) | -0.17** (0.07) | -0.32** (0.07) 
Holyoke -0.42** (0.10) | -0.39** (0.10) | -0.42** (0.10) | -0.39** (0.10) | -0.42** (0.10) | -0.39** (0.10) 
Lowell 0.02 (0.07) -0.06 (0.08) 0.02 (0.07) -0.06 (0.08) 0.02 (0.07) -0.06 (0.08) 
Lynn 0.08 (0.07) 0.02 (0.07) 0.08 (0.07) 0.02 (0.07) 0.08 (0.07) 0.02 (0.07) 
Springfield -0.07 (0.06) -0.15** (0.06) -0.07 (0.06) -0.15** (0.06) -0.07 (0.06) -0.15** (0.06) 
Worcester 0.15** (0.06) 0.05 (0.06) 0.15** (0.06) 0.05 (0.06) 0.15** (0.06) 0.05 (0.06) 
Constant (Bo) 0.09** (0.03) -0.01 (0.03) 0.09** (0.03) -0.01 (0.03) 0.09** (0.03) -0.01 (0.03) 
Time random-effects parameters 0.00 (0.01) 0.00 (0.01) 0.00 (0.01) 0.00 (0.01) 0.00 (0.01) 0.00 (0.01) 
School random-effects parameters 0.12 (0.34) 0.12 (0.34) 0.12 (0.34) 0.12 (0.34) 0.12 (0.34) 0.12 (0.34) 
Residual random-effects parameters 0.81 (0.90) 0.79 (0.89) 0.81 (0.90) 0.79 (0.89) 0.81 (0.90) 0.79 (0.89) 


Note. Standard errors are presented in parentheses. 


* p< .05. ** p< .01. 
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Table E2. Regression Outcomes for Special Populations Within SRG and Comparison 


Schools: Attendance Rate 


Enalsh Language | Fre/or Reduced | syeilEaucation 
Grant (B,) -0.006 (0.007) -0.006 (0.007) -0.005 (0.007) 
Time (B2) -0.001 (0.000) -0.001 (0.000) -0.001 (0.000) 
Grant x Time (£3) 0.002** (0.000) 0.002** (0.000) 0.002** (0.000) 


Grant x Post Year 1 (B,) 


0.001 (0.001) 


0.001 (0.002) 


-0.001 (0.001) 


Grant x Post Year 2 (Bs) 


-0.001 (0.001) 


-0.001 (0.002) 


-0.003* (0.001) 


Grant x Post Year 3 (Bs) -0.003 (0.002) 0.002 (0.003) -0.005** (0.002) 
Grant x Subgroup (B10) 0.000 (0.001) 0.000 (0.001) -0.004** (0.001) 
Say hacwpecta 0.002 (0.002) 0.001 (0.002) 0.008** (0.002) 
sea ory erences 0.002 (0.002) 0.000 (0.002) 0.006** (0.002) 
Say apeca! 0.002 (0.002) 0.002 (0.003) 0.007* (0.003) 
Year 2011 (Br) 0.002 (0.001) 0.002 (0.001) 0.002 (0.001) 


Year 2012 (Bs) 


0.005** (0.002) 


0.005** (0.002) 


0.005** (0.002) 


Year 2013 (Bo) 


0.006** (0.002) 


0.006** (0.002) 


0.006** (0.002) 


Special education 


-0.019** (0.000) 


-0.019** (0.000) 


-0.018** (0.000) 


Free- or reduced-price lunch 


-0.014** (0.000) 


-0.014** (0.000) 


-0.014** (0.000) 


English language learner 


0.010** (0.000) 


0.010** (0.000) 


0.010** (0.000) 


Hispanic -0.009** (0.000) -0.009** (0.000) -0.009** (0.000) 
Black 0.009** (0.000) 0.009** (0.000) 0.009** (0.000) 
Asian/other 0.015** (0.000) 0.015** (0.000) 0.015** (0.000) 
Brockton 0.003 (0.007) 0.003 (0.007) 0.003 (0.007) 
Fall River -0.009 (0.007) -0.009 (0.007) -0.009 (0.007) 
Holyoke -0.009 (0.010) -0.009 (0.010) -0.009 (0.010) 
Lowell 0.006 (0.008) 0.006 (0.008) 0.006 (0.008) 
Lynn 0.012 (0.007) 0.012 (0.007) 0.012 (0.007) 
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English Language Free- or Reduced- : F 
Learners Price Lunch Special Education 

Springfield 0.006 (0.006) 0.006 (0.006) 0.006 (0.006) 
Worcester 0.027** (0.006) 0.027** (0.006) 0.027%* (0.006) 
Constant (Bo) 0.941 ** (0.003) 0.941 (0.003) 0.941** (0.003) 
Time random-effects 0.000 (0.001) 0.000 (0.001) 0.000 (0.001) 
parameters 
School random-effects 0.001 (0.037) 0.001 (0.037) 0.001 (0.037) 
parameters 
Residual random-effects 0.006 (0.075) 0.006 (0.075) 0.006 (0.075) 
parameters 


Note. Standard errors are presented in parentheses. 


* p< .05. ** p< .01. 
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